Description of Changes

Comment pasted from within watch_maybe_launch_module_host:

try_init_host is not cancel safe, as it will spawn other async tasks
which hold a filesystem lock past when try_init_host returns or is cancelled.
This means that, if try_init_host is cancelled, subsequent calls will fail.

This is problematic because Axum will cancel its handler tasks if the client disconnects,
and this method is called from Axum handlers, e.g. for the subscribe route.
tokio::spawn a task to build the Host and install it in the guard,
so that it will run to completion even if the caller goes away.

Note that tokio::spawn only cancels its tasks when the runtime shuts down,
at which point we won't be calling try_init_host again anyways.

This fixes (hopefully) the dreaded replay-loop bug, which we belive was caused by Axum dropping the watch_maybe_launch_module_host future after it acquired the Lockfile and leaked that file to a non-cancel-able async task, leading subsequent calls to fail due to the Lockfile still being held. Now, any call to watch_maybe_launch_module_host which enters try_init_host will run to completion and install the new Host into the HostController, even if the calling future is dropped or cancelled.

Expected complexity level and risk

1: wrapping already-async stuff in tokio::spawn is a minimal change.

Testing

• Manual testing in a production-like cluster to see if this fixes the replay loop.